System and method of indicating the distance or the surface of an image of a geographical object

ABSTRACT

A system and method is provided that displays cursors for street level images, where the cursor changes appearance based on the objects in the image, such as the geographic distance between the objects and the camera position and the surface of the objects. For example, the cursor may appear to lie flat against the objects in the image change size based on the distance between the camera and object&#39;s surface.

BACKGROUND OF THE INVENTION

Services such as Google Maps are capable of displaying street levelimages of geographic locations. These images, known on Google Maps as“Street View”, typically comprise photographs of buildings and otherfeatures, and allow a user to view a geographic location from a person'sperspective as compared to a top-down map perspective.

When displayed on a monitor, a user may interact with an image bymanipulating a cursor and 3D objects displayed on the image. Forexample, Google Maps often draws a yellow line corresponding with astreet on the image; the line is drawn as if it were painted on thestreet. When the user moves the cursor to an arrow on the yellow line,and clicks the arrow, the image may change to a nearby street levelimage taken from a different perspective or location. However, eventhough the cursor may be used to change or otherwise interact with thestreet level image, the cursor itself does not convey information aboutthe street level image itself.

BRIEF SUMMARY OF THE INVENTION

In one aspect of the invention, a method of displaying images isprovided. The method includes: receiving an image representing ageographical object captured by a camera at a camera position; receivingsurface data representing a surface of the geographical object and thedistance of the surface relative to the geographical object; displayingthe image on an electronic display; receiving a screen position, wherethe screen position identifies a position on the image relative to thedisplay; selecting, with a processor, surface data depending on whetherthe screen position corresponds with the location of the surface withinthe image; determining, with a processor, a visual characteristic of acursor based on the selected surface data and the distance of thesurface; and displaying the cursor with the visual characteristic on theelectronic display.

Another aspect provides a method of displaying images that includes:receiving an image representing a geographical object captured by acamera at a camera position; displaying the image on an electronicdisplay; receiving distance data representing the distance between asurface of the geographical object and the camera position; receiving ascreen position, where the screen position identifies a position on theimage relative to the display; selecting, with a processor, distancedata depending on whether the screen position corresponds with thelocation of the surface within the image; determining, with a processor,a visual characteristic of a cursor based on the selected distance data;and displaying the cursor with the visual characteristic on theelectronic display.

Yet another aspect provides a method of displaying images that includes:receiving an image representing a geographical object captured by acamera at a camera position; receiving distance data representing thedistance between a surface of the geographical object and the cameraposition; displaying the image on an electronic display; receivingorientation data representing the orientation of the surface relative tothe camera angle; receiving a screen position, where the screen positionidentifies a position on the image relative to the display; selecting,with a processor, distance data and orientation data depending onwhether the screen position corresponds with the location of the surfacewithin the image; determining, with a processor, a visual characteristicof a cursor based on the selected distance data and orientation data;and displaying the cursor with the visual characteristic on theelectronic display.

Still another aspect provides a system having a variety of componentsincluding a user input device; a memory storing instructions, image datarepresenting geographic objects captured by a camera, and datarepresenting the position of those surfaces of the objects that arefacing the camera; a processor in communication with the user inputdevice so as to process information received from the user input devicein accordance with the instructions; and a display in communicationwith, and displaying information received from, the processor. Theinstructions include determining a position on the image based oninformation received from the user input device, determining the surfaceof the object at said position of the image, determining a shape andsize based on the data representing the position of said surfacerelative to distance to the viewpoint of the image, and displaying, onthe display, the determined shape at the determined size.

A further aspect relates to a method of method of displaying imagescomprising: receiving an image representing a plurality of geographicalobjects captured by a camera at a camera position; receiving distancedata representing the distance between the geographical objects and thecamera position; receiving a screen position, where the screen positionidentifies a position on the image relative to the display; displaying,on an electronic display, the image; selecting, with a processor,distance data associated with one of the plurality of geographicalobjects depending on whether the screen position corresponds with thelocation of the geographical object within the image; determining, witha processor, a visual characteristic of a cursor based on the selecteddistance data; and displaying the cursor with the visual characteristic.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional diagram of a system in accordance with an aspectof the invention.

FIG. 2 is a pictorial diagram of a system in accordance with an aspectof the invention.

FIG. 3 is a functional diagram of data representing the surfaces ofbuildings in a street level image.

FIG. 4 is a functional diagram of data representing the surfaces ofbuildings in a street level image.

FIG. 5 is a functional diagram of data representing the surfaces ofbuildings in a street level image.

FIG. 6 is a screen shot in accordance with an aspect of the invention.

FIG. 7 is a screen shot in accordance with an aspect of the invention.

FIG. 8 is a functional diagram of data representing the surfaces ofbuildings in the street level image of FIG. 7.

FIG. 9 is a screen shot in accordance with an aspect of the invention.

FIG. 10 is a screen shot in accordance with an aspect of the invention.

FIG. 11 is a screen shot in accordance with an aspect of the invention.

FIG. 12 is a screen shot in accordance with an aspect of the invention.

FIG. 13 is a screen shot in accordance with an aspect of the invention.

FIG. 14 is a flowchart in accordance with an aspect of the invention.

FIG. 15 is a flowchart in accordance with an aspect of the invention.

DETAILED DESCRIPTION

In one aspect, the system and method displays cursors for street levelimages, where the cursor appears to lie flat against the objects in theimage and changes based on the user's manipulation of a mouse, keyboardor other navigation method. The size of the cursor also changes based onthe distance of an object under the cursor to the camera position. Thisand other aspects are described below.

As shown in FIGS. 1-2, a system 100 in accordance with one aspect of theinvention includes a computer 110 containing a processor 210, memory 220and other components typically present in general purpose computers.

Memory 220 stores information accessible by processor 210, includinginstructions 240 that may be executed by the processor 210. It alsoincludes data 230 that may be retrieved, manipulated or stored by theprocessor. The memory may be of any type capable of storing informationaccessible by the processor, such as a hard-drive, memory card, ROM,RAM, DVD, CD-ROM, write-capable, and read-only memories. The processor210 may be any well-known processor, such as processors from IntelCorporation or AMD. Alternatively, the processor may be a dedicatedcontroller such as an ASIC.

The instructions 240 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. In that regard, the terms “instructions,” “steps” and“programs” may be used interchangeably herein. The instructions may bestored in object code form for direct processing by the processor, or inany other computer language including scripts or collections ofindependent source code modules that are interpreted on demand orcompiled in advance. Functions, methods and routines of the instructionsare explained in more detail below.

Data 230 may be retrieved, stored or modified by processor 210 inaccordance with the instructions 240. For instance, although the systemand method is not limited by any particular data structure, the data maybe stored in computer registers, in a relational database as a tablehaving a plurality of different fields and records, XML documents, orflat files. The data may also be formatted in any computer-readableformat such as, but not limited to, binary values, ASCII or Unicode.Moreover, the data may comprise any information sufficient to identifythe relevant information, such as numbers, descriptive text, proprietarycodes, pointers, references to data stored in other memories (includingother network locations) or information which is used by a function tocalculate the relevant data.

Although the processor and memory are functionally illustrated in FIG. 1within the same block, it will be understood by those of ordinary skillin the art that the processor and memory may actually comprise multipleprocessors and memories that may or may not be stored within the samephysical housing. For example, some of the instructions and data may bestored on removable CD-ROM and others within a read-only computer chip.Some or all of the instructions and data may be stored in a locationphysically remote from, yet still accessible by, the processor.Similarly, the processor may actually comprise a collection ofprocessors which may or may not operate in parallel.

In one aspect, computer 110 is a server communicating with one or moreclient computers 150, 170 (only client 150 being shown in FIG. 1 forclarity). Each client computer may be configured similarly to the server110, with a processor, memory and instructions. Each client computer150, 170 may be a personal computer, intended for use by a person190-191, having all the internal components normally found in a personalcomputer such as a central processing unit (CPU), display device 160(for example, a monitor having a screen, a projector, a touch-screen, asmall LCD screen, a television, or another device such as an electricaldevice that is operable to display information processed by theprocessor), CD-ROM, hard-drive, user input (for example, a mouse 163,keyboard, touch-screen or microphone), speakers, modem and/or networkinterface device (telephone, cable or otherwise) and all of thecomponents used for connecting these elements to one another. Moreover,computers in accordance with the systems and methods described hereinmay comprise any device capable of processing instructions andtransmitting data to and from humans and other computers includinggeneral purpose computers, PDAs, network computers lacking local storagecapability, and set-top boxes for televisions.

Although the client computers 150 and 170 may comprise a full-sizedpersonal computer, many aspects of the system and method areparticularly advantageous when used in connection with mobile devicescapable of wirelessly exchanging data with a server over a network suchas the Internet. For example, client computer 170 may be awireless-enabled PDA such as a Blackberry phone or an Internet-capablecellular phone. In either regard, the user may input information using asmall keyboard (in the case of a Blackberry phone), a keypad (in thecase of a typical cell phone), a touch screen (in the case of a PDA) orany other means of user input.

Client computers 150 and 170 may include a component, such as circuits,to determine the geographic location of the device. For example, mobiledevice 170 may include a GPS receiver 155. By way of further example,the component may include software for determining the position of thedevice based on other signals received at the mobile device 150, such assignals received at a cell phone's antenna from one or more cell phonetowers if the mobile device is a cell phone.

The server 110 and client computers 150 and 170 are capable of directand indirect communication, such as over a network 295. Although only afew computers are depicted in FIGS. 1-2, it should be appreciated that atypical system can include a large number of connected computers, witheach different computer being at a different node of the network 295.The network, and intervening nodes, may comprise various configurationsand protocols including the Internet, World Wide Web, intranets, virtualprivate networks, wide area networks, local networks, private networksusing communication protocols proprietary to one or more companies,Ethernet, WiFi and HTTP. Such communication may be facilitated by anydevice capable of transmitting data to and from other computers, such asmodems (e.g., dial-up or cable), networks and wireless interfaces.Server 110 may be a web server.

Although certain advantages are obtained when information is transmittedor received as noted above, other aspects of the system and method arenot limited to any particular manner of transmission of information. Forexample, in some aspects, information may be sent via a medium such as adisk, tape or CD-ROM. In other aspects, the information may betransmitted in a non-electronic format and manually entered into thesystem. Yet further, although some functions are indicated as takingplace on a server and others on a client, various aspects of the systemand method may be implemented by a single computer having a singleprocessor.

Map database 270 of server 110 stores map-related information, at leasta portion of which may be transmitted to a client device. For example,map database 270 may store map tiles 272, where each tile is a map imageof a particular geographic area. Depending on the resolution (e.g.,whether the map is zoomed in or out), one tile may cover an entireregion, such as a state, in relatively little detail. Another tile maycover just a few streets in high detail.

The map information is not limited to any particular format. Forexample, the images may comprise street maps, satellite images, or acombination of these, and may be stored as vectors (particularly withrespect to street maps) or bitmaps (particularly with respect tosatellite images). The various map tiles are each associated withgeographical locations, such that the server 110 is capable ofselecting, retrieving and transmitting one or more tiles based on areceipt of a geographical location, which may be a single point, arange, or some other indication.

The map database may also store street level images 274. Street levelimages comprise images of objects captured by cameras at particulargeographical locations in a direction roughly parallel to the ground.For example, a single street level image may show a perspective view ofa street and its associated buildings, taken at a position a few feetabove the ground (e.g., from a camera mounted on top of a vehicle, andat or below the legal limit for typical vehicles in certain states(e.g., 7-14 feet)) and in a direction generally parallel to the ground(e.g., the camera view was pointed down the street into the distance).Street level images are not limited to any particular height above theground, for example, a street level image may be taken from the top ofbuilding.

In one aspect of the system and method, the street level images arepanoramic images, such as 360° panoramas centered at the geographiclocation associated with the image. The panoramic street-level viewimage may be created by stitching together a plurality of photographsrepresenting different perspectives from a geographical vantage point.In other aspects, only a single street level image pointing in aparticular direction may be available at any particular geographicallocation. The street level images are thus typically associated withboth a geographical location and information indicating the orientationof the image. For example, each image may be associated with both alatitude and longitude, and data that allows one to determine whichportion of the image corresponds with facing north, south, east, west,northwest, etc.

Street level images may also be stored in the form of videos, such as bydisplaying MPEG videos captured by an analog video camera or displaying,in succession, time-sequenced photographs that were captured by adigital still camera.

Memory 220 may also store surface information 276 describing the outersurface of objects in the street level images. By way of example only,each street level image may be associated with information describingthe proximity of the objects' surface to the camera. For example, FIG. 3shows two buildings 310 and 320 that may appear in a street level image.In that regard, each surface may be represented as collection of points.Each point, in turn, may be represented as a vector, whereby each pointis stored with respect to its distance to the camera, and its angle withrespect to the direction in which the camera is pointed. Thus, point 350of surface 311 of building 310 may be defined by its distance 370 fromthe camera and its angles along the ground 372 (the x-y direction) andheight 374 (the z-direction) relative to the direction 380 in which thecamera 390 is pointed. Such information may be collected by using alaser range finder in combination with the camera taking the streetlevel images.

Although some formats may be more advantageous than others, the systemand method is not limited to any particular format of storing thesurface information. For example, as shown in FIG. 4, if the latitude,longitude and altitude of the camera 490 are known, the surface of 411of building 410 may be stored as collection of points (shown as blackdots) each having an associated latitude, longitude and altitude. Thesurface data may thus represent a cloud of 3D points positioned in spacerelative to a reference point.

Such surface data may be sent from a server to a client as a grid ofdiscrete values, where each element of the grid corresponds with a pixelof a street level image. If so, the value of the depth map at each pixelmay represent the distance from the camera position to the portion ofthe geographic object shown at that pixel.

Yet another format stores the objects in the street level image as 3Dmodels. By way of example only and shown in FIG. 5, rather than storingcollections of points, the facades and other surfaces of the buildingsmay be represented as rectangles, triangles or other shapes defined byvertices having positions in space, such as latitude, longitude andaltitude. Thus, front facade 511 of building 510 may be represented as arectangular plane having four points 551-554, with each point defined inan (x,y,z) format (such as latitude, longitude, altitude). Similarly,side facade 512 may be represented as a rectangular plane defined byfour points 553-556, two of which are shared with the front facade 511.Accordingly, the buildings and other objects that may be present in astreet level image may also be stored as 3D models comprising polygons.

Still another format stores the objects as a set of planes correspondingwith the object surfaces facing the camera that captured the streetlevel image. Each plane may be associated with a unique index number,and the vertex of each plane may be defined in an (x,y,z) format (suchas latitude, longitude, altitude). The data defining the planes may alsoassociate each pixel of the street level image with one of the planes.Thus, instead of defining the distance between the camera and the objectrepresented at the pixel, the value would represent the index of theplane representing the object surface at that pixel. Representingsurfaces in this fashion may permit a processor to quickly retrieve anddetermine the position and orientation of each surface at each pixel ofa street level image. Pixels that are not associated with a surface maybe associated with a null or default surface value.

Many of the formats permit the surface information to be storedindependently of the street level images taken by the camera. As shownin FIG. 5, if the building surfaces are stored as 3D models relative tothe latitude and longitude of the earth, the data associated with the 3Dmodel of building 510 does not change regardless of whether the cameraview of the building is at first position 591 or second position 592.Accordingly, in some formats, the surface information may be storedwithout regard to a particular street level image.

A variety of systems and methods may be used to collect the surfaceinformation. By way of example only and as noted above, a laser rangefinder may be used. In addition, stereoscopic systems employing twovideo cameras, spaced slightly apart yet looking at the same scene, maybe used as well; by analyzing the slight differences between the imagesseen by each camera, it is possible to determine the distance at eachpoint in the images. In yet another aspect, the information may becompiled by using a single video camera, travelling at a particularvelocity, to capture the street level imagery as the scenery passes by.The video may not only be used and shown as the street level image, butsubsequent frames may be compared to extract the different distancesbetween the objects and the camera (e.g., mountains in the distance willstay in the frame much longer than a fire hydrant passing by along thestreet).

In addition to the operations illustrated in FIGS. 14-15, variousoperations in accordance with a variety of aspects of the invention willnow be described. It should be understood that the following operationsdo not have to be performed in the precise order described below.Rather, various steps can be handled in reverse order or simultaneously.

FIG. 6 illustrates a screen shot of a map displayed by the displaydevice at the client computer from a top-down perspective. For example,the system and method may be implemented in connection with an Internetbrowser such as Google Chrome (not shown) displaying a web pagecontaining a map 665 and other information. The program may provide theuser with a great deal of flexibility when it comes to both identifyinga location to be shown in a street level view and requesting the streetlevel image. For example, the user may enter information such as anaddress, the name of building, latitude and longitude, or some otherinformation that identifies a particular geographical location in textbox 610. The user may further use a mouse or keypad to move a cursor 660to identify the particular geographical location of the street levelimage. Yet further, the program may provide a button 670 or some otherfeature that allows a user to request a street level view at thespecified geographical location.

Upon the user requesting an image from a street level viewpoint at aparticular geographical location, server 110 retrieves the appropriatestreet level image based on the requested location. For example, if thestreet level images are stored based on the latitude/longitudecoordinates of the camera that captured the image, the closest image tothe requested latitude/longitude will be retrieved. The street levelimage is displayed as shown in FIG. 7 on the display device 160 ofclient computer 170. As noted above, the image may be a bitmap of apicture taken by a camera at the requested geographical location. Asshown in FIG. 7, the image may show only one direction, such as north,east, southwest, etc. Alternatively, the image may show a portion of a360° panoramic image wrapping around the geographical location, such asthe portion of the panoramic image that corresponds with the desiredorientation.

FIG. 7 illustrates just one possible street level image 765 representinggeographic objects such as buildings, walls, streets, and lamp posts.Any other objects at geographic locations may also be represented by theimage data.

As shown in FIG. 7, the street level image 765 may be shown in thebrowser along with controls 750 for zooming the image and controls 755for changing the location or direction of the viewpoint (such as thecamera angle). In accordance with the instructions, operation ofcontrols 750 or 755 will change the image 765 being displayed by usingthe information that was already received from the server (such as byusing a different portion of the received image if an entire 360°panorama was received) or retrieving additional information (such as byrequesting another image if the user wants to view a scene looking eastfrom the location and the received image only shows the scene lookingnorth from the location). Other navigation controls may be included aswell, such as panning controls in the form of arrows disposed along thestreet. Such arrows may be selected by a user (by clicking or bydragging along the street line) to change the vantage point from up ordown the street.

The display device may also display a cursor. As is well known, userscan identify positions on a screen by operating user devices such asmoving a computer mouse, pressing arrow keys on a keyboard, or tapping atouch-sensitive screen. By way of example, the mouse may provideinformation indicating how much the position should change and atouch-sensitive screen may provide information regarding what theposition should be. Hereafter, references to “screen position” means aposition relative to the display device. By way of example only, thescreen position of a monitor may be a position on the screen of themonitor relative to the screen's top-left corner. The screen position ona PDA with a fixed screen may be a position on a touch-screen of a PDArelative to the center of the device. Yet further, a screen position mayrefer to a position on a movie screen that is displaying imagesprojected by a projector, or any other location on any other articlethat is capable of displaying information that changes in response toinstructions.

If the user uses a mouse to select the screen position, the selectedposition may be identified by displaying a mouse cursor 780 at theparticular screen position. A display may show more than one cursor atmore than one screen position. For example, a display may simultaneouslydisplay both an arrow-shaped mouse cursor identifying a screen positionselected by a mouse, and an I-beam shaped text cursor identifying ascreen position selected by a keyboard.

In one aspect of the invention, surface information 276 is retrievedalong with the street level image. For example, if the surfaceinformation of objects in the street level image view 765 are stored as3D models comprised of multiple polygons (hereafter, “object models”) asfiguratively shown in FIG. 8, the object models 810, 820, 830-833 withina particular distance of the geographical location 890 of the streetlevel image are sent to the client. Optionally, only those objects thatlie in the direction of the camera direction 891 may be transmitted,such as building models 810, 820 and 830 and wall 831. (Though notintended to be to scale, models 810, 820 and 830 correspond withbuildings 910, 920 and 930 in FIG. 9.)

As shown in FIG. 9, a user may move a mouse cursor 990 across the streetlevel image 965 by manipulating the mouse associated with the clientcomputer (or by manipulation of other user input devices).

In one aspect of the system and method, the processor displays a cursorwhose position on the street level image corresponds with auser-selected screen position and whose shape depends on the position ofthe objects shown in the street level image.

For example, when positioned over an object (such as a building) in thestreet level image, the object-dependant cursor may be a circle whosediameter changes based on object's distance from the camera. As shown byexample in FIG. 9, if the user has moved mouse cursor 990 to a screenposition that is over building 910, the processor may display a circle991 at the same screen position. The diameter of the circle may be 30pixels wide.

When the user moves the mouse cursor over another object, such asbuilding 920, the diameter of the circle will change if the object iscloser or farther away than building 910. For example, when the screenposition of the mouse cursor is moved in direction 955 from the facadeof building 910 to building 920, which is behind building 920 based onthe current camera position and angle, the diameter of the circularobject-dependant cursor may change from 30 pixels to 15 pixels (asrepresented by circle 992). The smaller size indicates that the newlypointed-to object is further away. If the screen position is changed yetagain to where building 930 is being displayed on the screen, the cursormay again change size again to 20 pixels (assuming the front of building930 is closer to the camera in the street level image than building 920but farther than building 910). The size changes may be proportionallyassociated, directly or exponentially or otherwise, with the changes inthe distance from the camera position or other viewpoint from which theobjects are currently being viewed.

The system and method is not limited to any particular method ofdetermining the closest object surface that corresponds with the screenposition within the displayed street level image. For example, if thesurface information is expressed in terms of 3D objects, the closestsurface may be determined by use of ray tracking. The length of the raywould indicate the size of the object-dependant cursor, e.g., the longerthe ray the smaller the cursor. Ray tracing may be particularlyeffective when used to determine the surfaces of objects at the poles ofspherical panoramas. Other methods, such as scan conversion andz-buffering, may also be used. Yet further, if the surface data is sentas a 2-D array of distance values for each pixel, the appropriatedistance data may be quickly retrieved by determining the position ofthe pixel in the image that is being pointed at by the mouse cursor.

In yet another aspect, the shape of the object-dependant cursor may bedeformed to appear as if it is lying on the surface. For example, asshown in FIG. 10, if building side 1022 is not squarely facing thecamera but rather angled towards it, and if the circle-shaped cursor isintended to give the impression of lying on the side of the building,the cursor will not be drawn as a circle (as far as the boundaries ofthe screen are concerned). Rather, to convey the impression of lyingagainst the building, the shape of the cursor is angled (to conform withthe angle of the building) and stretched (to conform with the buildingstretching into the distance).

In that regard, in one aspect of the invention, circle-shapedobject-dependant cursors may be shown as stretched ellipses 1090 whenpointing to surfaces that are angled with respect to the camera. Othershapes could be similarly deformed to appear as if they are lying flaton an angled surface, such as drawing square-shaped object-dependantcursors on surfaces that directly face the camera and drawingtrapezoid-shaped object-dependant cursors on surfaces that have anorientation angling away from the camera and into the distance.

The object-dependant cursor may also wrap around objects. As shown inFIG. 11, the width and screen position of the object-dependant cursor1180 may cause it to overlap with two different surfaces, namely front1131 and side 1132 of building 1130. Accordingly, a portion of thecursor 1180 may be drawn flat against the front 1131 and another portionagainst the side 1132 of the building. Again, in this aspect, it isintended that the object-dependant cursor appear to lie flat against theobject shown in the street level image.

Although the buildings shown in FIG. 8 are modeled as simple box-shapedobjects with rectangular sides—and the example object-dependant cursorsof FIGS. 9-11 reflect these simple models—the models may be far morecomplex. They may include indentations (such as windows), extensions(such as balconies and water towers), and non-rectangular shapes (suchas triangles and curves). Accordingly, in some aspects of the system andmethod, the object-dependant cursor will wrap in and around suchwindows, balconies, towers, etc. Moreover, the surface data used toconfigure the shape of the object-dependant cursor may be kept simple(for aesthetic or processing reasons) even if the surfaces of thebuildings displayed in the actual street level image are a complexcollection of indentations, protrusions and curves.

The system and method may also ignore certain surfaces for the purposeof shaping and sizing the cursor. For example, the surface of thelamppost 1190 in FIG. 11 may be too small to provide meaningfulinformation for the purpose of drawing the cursor. Accordingly, if themouse cursor is over the lamppost, the system and method may ignore thelamppost for the purpose of determining the visual characteristics ofthe object-dependant cursor and use the surface behind it instead, suchas building wall 1195. Objects deemed too small or otherwiseinappropriate for the rendering the object-dependant cursor may befiltered by setting the object's depth/position information equal tothat of the object behind it. For example, if the surface information276 is stored as the distance from the camera position to the object ateach pixel of the street level image, surfaces that are only a fewpixels wide may have their values changed to the same distance as thesurrounding pixels. Pattern matching may also be used to identifysurfaces associated with other objects, such as cars and people.

In yet another aspect, the system and method does not wrap the objectaround the edge of an object. Rather, as shown in FIG. 13, the systemand method may determine the characteristics of the object-dependantcursor based solely on the point below the mouse cursor. In that regard,the object-dependant cursor 1310 may hang over the building 1320 if themouse cursor 1330 points at the edge of the building 1320.

If no surface is found at the particular screen position, the shape ofthe object-dependant cursor may change accordingly, such as showing amouse cursor but no object-dependant cursor (as shown in FIG. 7) orshowing a different shape (such as a question mark).

Yet further, the mouse cursor may not be shown at all, such that onlythe object-dependant cursor is shown. In addition, the location of theobject-dependant cursor does need not need to correlate with the mousecursor's screen position at all times. For example, the calculationsrequired to display the object-dependant cursor may cause the display ofthe object-dependant cursor to lag behind the mouse cursor. Moreover, inone aspect, the object-dependant cursor does not automatically move withthe mouse cursor. Instead, the object-dependant cursor may move when theuser requests, such as by clicking a button of the mouse, or wheneverthe screen position of the mouse cursor has hovered over the same screenposition for some period of time.

The object-dependant cursor may also be moved independently of the useroperating user input devices. For example, the cursor may be moved inaccordance with a predefined program or in a manner to highlight thesurfaces.

The shape is not limited to circles and changing the size. For example,the shape may be a square, oval, star or any other shape.

Moreover, while changing the size of the cursor relative to the screensize may be particularly advantageous, other embodiments may includeother changes such as a change in shape (from circle to rectangle) orcolor (from light yellow to dark yellow). Additionally, the shape,color, and configuration of the pancake cursor may be updated inreal-time. The object-dependant cursor may also be somewhat transparent,with the transparency being determined based on information associatedwith the objects in the street level image.

In yet another aspect, some of the characteristics of theobject-dependant cursor are intended to convey information unrelated tothe surfaces or distance of the objects shown in the street level image.For example, changing the color to red or the shape to a magnifyingglass may indicate that the map database stores a street level imagewith a camera position closer to the building under the object-dependantcursor than the camera position of the street level image currentlybeing displayed. The user could then click the building to move closer.

In that regard, the system and method may include cursors having somevisual characteristics that are dependant on the distance or surface ofthe objects shown in the street level images, some visualcharacteristics that are independent of such objects but dependant oninformation contained in the map database, and yet other visualcharacteristics that are independent of both such objects and database.

In one aspect, the shape of the cursor, or another visual characteristicof the cursor, reflects the nature of the geographic object under thecursor. For example, as shown in FIG. 12, the cursor may have arectangular shape 1210 when it is on building 1230 in the street levelimage 1265, but a circular shape 1220 when it is moved onto street 1250in the same image.

The shape may also reflect the orientation of the surface of the object.In that regard, a rectangle may be used when the surface of the objectunder the cursor is generally orthogonal to the ground (e.g., abuilding). In contrast, a circle may be used when the surface of theobject under the cursor is generally parallel to the ground (e.g., astreet).

Yet further, the shape of the cursor may indicate what will happen uponuser activation (such as by clicking the mouse or hitting the enterkey). For example, if the street level image is oriented so that it islooking down the street with buildings on the side, clicking on abuilding may change the orientation of the viewpoint so that a streetlevel image is shown facing the building. However, clicking on a streetmay change the position of the viewpoint (such as down the street)without changing the orientation. In that regard, a rectangular shape,with the bottom of the rectangle oriented parallel to the street, may beused to indicate the change in orientation, whereas a circular shape mayindicate that the orientation will not change.

Yet further, the shape may reflect multiple conditions, such as both thetype of geographic object and what will happen upon user activation.

Still another aspect of the system and method provides information thatassociates other street level images with the information contained inthe street level image being displayed. Server 110 may thus provideclient computer 150 with data defining, for each pixel of the streetlevel image, the identity of a different street level image 274 to bedisplayed when that pixel is selected. For example, the data mayidentify one street level image to be shown when a building in thedistance is clicked and yet another street level image to be shown whena nearby bridge is clicked. This information may be calculated by theserver 110 in advance of a request by a user 190, thus allowing the nextstreet level image to be more rapidly determined than by (1) determiningthe latitude/longitude position of the object in the image that wasclicked, (2) determining the closet street level image to thatlatitude/longitude position, and (3) displaying the street level image.

In one aspect of the system and method, the cursor has a minimum andmaximum size. For example, the minimum cursor size may be determined asthe smallest visually acceptable size for indicating the surface of anobject at the furthest point in the image. Similarly, the largest cursorsize may be determined as the largest visually acceptable size forindicating the surface of the surface at the nearest point in the image.Thus, as the user selects locations starting from the nearest locationto the furthest location in the image, the size of the cursor mayrespectively transition from the largest size to the smallest size.

In still another aspect, the cursor changes based on the distance to aposition of the object that does not correspond to its camera-facingsurfaces, such as a building's center.

In one aspect, the size of the object-dependant cursor may be selectedby the user. By way of example, the user may click and drag over acontiguous area of the street level image, in which case theobject-dependant cursor is sized to correspond with the dragged area.

The object-dependant cursor may also be offset relative to the screenposition of the mouse cursor. For example, if the street level image isdisplayed on a touch-screen, the object-dependant cursor may be obscuredby the user's finger. Accordingly, the object-dependant cursor mayreflect the surface of an object, and be displayed, at a point that isabove the mouse cursor, e.g., a few pixels above the point at which theuser is touching the screen.

Most of the foregoing alternative embodiments are not mutuallyexclusive, but may be implemented in various combinations to achieveunique advantages. As these and other variations and combinations of thefeatures discussed above can be utilized without departing from theinvention as defined by the claims, the foregoing description of theembodiments should be taken by way of illustration rather than by way oflimitation of the invention as defined by the claims. It will also beunderstood that the provision of examples of the invention should not beinterpreted as limiting the invention to the specific examples; rather,the examples are intended to illustrate only one of many possibleembodiments.

The invention claimed is:
 1. A method of displaying images comprising:receiving, by one or more computing devices, a street-level imagerepresenting a geographical object captured by a camera at a cameraposition; receiving, by the one or more computing devices, surface datarepresenting a surface of the geographical object and the distance ofthe surface from the camera position, the surface data including datathat represents the surface of the geographical object based on athree-dimensional coordinate system; displaying, by the one or morecomputing devices, the street-level image on an electronic display;receiving, by the one or more computing devices, a screen position,where the screen position identifies a position on the street-levelimage relative to the display at which to display a cursor; selecting,by the one or more computing devices, surface data depending on whetherthe screen position corresponds with the location of the surface of thegeographic object depicted within the street-level image; determining,by the one or more computing devices, a visual characteristic of thecursor based on the selected surface data, an orientation of the surfacerelative to the camera, and the distance of the surface to the camera;and displaying, by the one or more computing devices, the cursor withthe visual characteristic on the electronic display to provide theappearance of the cursor conforming to the surface of the geographicalobject within the street-level image such that the shape of the cursorappears to lie flat against the surface of the geographical object,wherein the visual characteristic of the cursor is configured to changebased on changes in the received screen position and correspondingchanges in the selected surface data, an orientation of the surfacerelative to the camera, and the distance of the surface to the camera,and wherein a shape of the cursor is dependent on the selected surfacedata of the geographical object.
 2. The method of claim 1, wherein theangle of the camera is substantially parallel to the ground.
 3. Themethod of claim 1 wherein the shape of the cursor is generally circularif the surface data indicates the surface is generally facing the cameraand generally elliptical if the surface data indicates the surface isangled away from the camera.
 4. The method of claim 1 wherein the visualcharacteristic of the cursor depends on the type of the surface.
 5. Themethod of claim 4 wherein a first type of surface is avertically-oriented surface and a second type of surface is ahorizontally-oriented surface.
 6. The method of claim 1 wherein thevisual characteristic of the cursor is its color.
 7. The method of claim1 wherein the screen position depends on the position of a mouse cursor.8. The method of claim 1 wherein: the street-level image represents aplurality of geographical objects captured by the camera; the surfacedata represents a plurality of surfaces; and the surface data isselected depending on whether the screen position corresponds with thelocation of one of the plurality of surfaces within the street-levelimage.
 9. A method of displaying images comprising: receiving, by one ormore computing devices, an image representing a geographical objectcaptured by a camera at a camera position at or above street level, theimage including data that represents the geographical object based on athree-dimensional coordinate system; displaying, by the one or morecomputing devices, the image on an electronic display; receiving, by theone or more computing devices, a plurality of different screen positionsrepresenting a command to move a cursor in the image between a firstposition overlapping the geographical object and a second position notoverlapping the geographical object; and displaying, by the one or morecomputing devices, the cursor with a first visual characteristic on theelectronic display when overlapping the geographical object anddisplaying, by the one or more computing devices, the cursor with asecond visual characteristic when not overlapping the geographicalobject, wherein: the first visual characteristic represents a distancebetween the geographical object and the camera, wherein the first visualcharacteristic is configured to change based on changes in the distancebetween the geographical object and the camera at different screenpositions; and the first visual characteristic of the cursor depends onthe type of the geographical object, wherein the first visualcharacteristic is configured to change based on changes in the type ofgeographical object at different screen positions, and wherein a shapeof the first visual characteristic of the cursor is dependent on thegeographical object such that the shape of the cursor appears to lieflat against a surface of the geographical object.
 10. The method ofclaim 9, wherein the angle of the camera is substantially parallel tothe ground.
 11. The method of claim 9, wherein the first visualcharacteristic of the cursor is its size.
 12. The method of claim 9,wherein the first visual characteristic of the cursor is its color. 13.The method of claim 9, wherein the screen position depends on theposition of a mouse cursor.
 14. The method of claim 9, wherein thedistance data represents, for each of a plurality of pixels of theimage, the distance from the camera position to the surface of theobject represented in the image at the pixel.
 15. The method of claim 9,wherein the distance data represents a plurality of polygons havingvertices at positions, the positions of the vertices being associatedwith the geographical position of a surface of the geographical object.16. The method of claim 15 wherein the vertices are stored as valuesrepresenting the latitude, longitude and altitude of the surface. 17.The method of claim 15 wherein the vertices are stored as valuesrepresenting distances from the camera position.
 18. The method of claim9, wherein: the distance data represents the distances of a plurality ofsurfaces captured in the image; the distance data is selected dependingon whether the screen position corresponds with the location of one ofthe plurality of surfaces within the image; the first visualcharacteristic is the size of the cursor; and the size of the cursor ata first one of the plurality of surfaces relative to a second one of theplurality of surfaces is proportional to the difference in distancebetween the first surface and the camera position and the distancebetween the second surface and the camera position.
 19. A method ofdisplaying images comprising: receiving, by one or more computingdevices, a street-level image representing a geographical objectcaptured by a camera at a camera position; receiving, by the one or morecomputing devices, distance data representing the distance between asurface of the geographical object and the camera position, the distancedata including a two-dimensional array of distance values for each pixelof the received street-level image that represents the geographicalobject; displaying, by the one or more computing devices, the image onan electronic display; receiving, by the one or more computing devices,orientation data representing the orientation of the surface relative tothe camera angle; receiving, by the one or more computing devices, ascreen position, where the screen position identifies a position on thestreet-level image relative to the display; selecting, by the one ormore computing devices, distance data and orientation data depending onwhether the screen position corresponds with the location of the surfacewithin the street-level image, wherein selecting the distance dataincludes selecting at least one distance value from the two-dimensionalarray based on the screen position; determining, by the one or morecomputing devices, a visual characteristic of a cursor based on theselected distance data and orientation data; and displaying, by the oneor more computing devices, the cursor with the visual characteristic onthe electronic display and overlapping the street-level image, whereinthe visual characteristic of the cursor is configured to change based onchanges in the received screen position and corresponding changes in thedistance data and orientation data, and wherein a shape of the cursor isdependent on the selected data associated with the geographical objectsuch that the shape of the cursor appears to lie flat against thesurface of the geographical object.
 20. The method of claim 19, whereinthe angle of the camera is substantially parallel to the ground.
 21. Themethod of claim 19 wherein the visual characteristic comprises the sizeand shape of the cursor, wherein the shape is selected so as to conveythe appearance of resting on the surface, and wherein the size isdetermined to be indicative of the distance between the camera positionand the surface.
 22. The method of claim 19 wherein the visualcharacteristic of the cursor depends on the type of the object at thescreen position.
 23. The method of claim 19 wherein the screen positionis determined based on the position of a mouse cursor.
 24. A systemcomprising: a user input device; a memory storing instructions, imagedata representing buildings captured by a camera, and data representingthe position of those surfaces of the buildings that are facing thecamera, wherein the data representing the position of those surfacesrepresents the position of those surfaces based on a three-dimensionalcoordinate system; a processor in communication with the user inputdevice so as to process information received from the user input devicein accordance with the instructions; and a display in communicationwith, and displaying information received from, the processor; theinstructions comprising: instructions for determining a position on theimage based on information received from the user input device,instructions for determining the surface of the building at saidposition of the image, instructions for determining a shape and size ofa cursor based on the data representing the position of said surfacerelative to distance to the viewpoint of the image, and instructions fordisplaying, on the display, the cursor having the determined shape atthe determined size and overlapping the surface of the building, whereinthe shape and size of the cursor are configured to change based onchanges in the determined data representing the position of said surfacerelative to distance to the viewpoint of the image, and wherein a shapeof the cursor is dependent on the surface of the building such that theshape of the cursor appears to lie flat against the surface of thegeographical object.
 25. The system of claim 24 wherein the user inputdevice is a mouse.
 26. The system of claim 24 wherein the size dependson the distance between said surface and the position of the camera whenthe image was captured.
 27. The system of claim 26 wherein the image iscomprised of pixels and wherein the data representing the position ofthe surfaces comprises a value associated with a plurality of thepixels.
 28. The system of claim 26 wherein each value represents thedistance from the camera position to the surface of the object portrayedat the pixel.
 29. The system of claim 24 wherein the shape of the cursordepends on the orientation of the surface relative to the camera angle.30. The system of claim 24 wherein the determined shape is displayedproximate to the position on the image that was determined based on theinformation received from the user input device.
 31. The system of claim24 further comprising a network, a server storing the image and datarepresenting the surface positions, wherein the memory receives theimage and data from the server and over the network.
 32. The system ofclaim 24 wherein the image is a still image.
 33. The system of claim 24wherein the image is a video.
 34. The system of claim 24 wherein thedata representing the position of the surfaces represents a plurality ofpoints on the surface of the building.
 35. The system of claim 34wherein each point is associated with a distance and angle relative tothe camera position.
 36. The system of claim 34 wherein each point isassociated with a latitude, longitude, and altitude.
 37. The system ofclaim 24 wherein the data representing the position of the surfacesrepresents a plurality of polygons having vertices at locationscorresponding with the surfaces.
 38. A method of displaying imagescomprising: receiving, by one or more computing devices, a street-levelimage representing a plurality of geographical objects captured by acamera at a camera position; receiving, by the one or more computingdevices, distance data representing the distance between thegeographical objects and the camera position, the distance dataincluding a two-dimensional array of distance values for each pixel ofthe received street-level image that represents a geographical object;receiving, by the one or more computing devices, a cursor position,where the cursor position identifies a position on the street-levelimage at which to display a cursor, the cursor position corresponding toa surface of a first geographic object of the plurality of geographicalobjects; displaying, by the one or more computing devices, on thedisplay, the street-level image; selecting, by the one or more computingdevices, distance data associated with the surface of the firstgeographical object, wherein selecting the distance data includesselecting at least a first distance value from the two-dimensional arraybased on the cursor position; determining, by the one or more computingdevices, a visual characteristic of the cursor based on the selecteddistance data; displaying, by the one or more computing devices, in thestreet-level image, the cursor with the visual characteristic so thatthe cursor conforms to the surface of the first geographic object in thestreet-level image, wherein a shape of the cursor is dependent on theselected data associated with the surface of the first geographicalobject such that the shape of the cursor appears to lie flat against thesurface of the geographical object; receiving, by the one or morecomputing devices, an updated cursor position, the updated cursorposition identifying a second position on the street level image atwhich to display the cursor, the second cursor position corresponding toa surface of a second geographical object of the plurality ofgeographical objects; selecting, by the one or more computing devices,second distance data associated with the surface of the secondgeographical object, wherein selecting the second distance data includesselecting at least a second distance value from the two-dimensionalarray based on the updated cursor position; determining, by the one ormore computing devices, a different visual characteristic of the cursorbased on the second distance data; and displaying, by the one or morecomputing devices, the cursor in the street-level image with thedifferent visual characteristic so that the cursor conforms to thesurface of the second geographical object in the street-level image.